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STANDARDS COMPLIANT WATERMARKING FOR ACCESS 
MANAGEMENT 

Inventors: Viresh Ratnakar 
Onur G. Guleryuz 

5 CONTINUING APPLICATION DATA 

This application claims the benefit of U.S. Provisional Application No. 
60/307,046 filed July 20, 2001 under 35 U.S.C. § 119(e). 

BACKGROUND OF THE INVENTION 

Field of the Invention 

lO This invention relates to standards- or format-compliant visible 

watermarking techniques for digital images and video which operate on decoded 
transform-coefficients. The techniques can be implemented in apparatuses, 
methods, and programs of instructions, e.g., software. 

I Description of the Related Art 

15 A watermark is a digital pattern embedded into the original content of a host 

signal, such as a digital image. A watermark can be inserted into a digital image 
by manipulating pixel values in the spatial domain representation or by altering 
certain transform coefficients in the fi-equency (e.g., DOT) domain representation. 
In the case of a block-based DOT representation, such as that used in the JPEG or 

20 MPEG compression standards, fi-equency coefficients in select blocks can be altered 
to produce a desired watermark pattern. 

There are different types of watermarks which serve different purposes. One 
type of watermark is visibly appHed to a digital image or video fi:ame(s) to prevent 
or discourage unauthorized use of the image or video. The watermark may be 

25 something simple and unobtrusive, such as a small logo in a corner of the image or 
video frame. However, such a watermark can be easily cropped away or concealed 
without a drastic loss in quahty. At the other extreme, the entire image may be 
"corrupted" in a manner that can be undone using a key. This is similar to 
encrypting all of the data, except that straight-forward encryption does not leave 

30 the data format-compliant. 
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OBJECT AND SUMMARY OF THE E»JVENTION 

Object of the Invention 

It is therefore an object of the present invention to provide watermarking 
techniques that address a visible watermarking problem that lies between the two 
5 above-mentioned extremes, and provides a range of solutions for still image and 
video. 

Summary of the Invention 

According to one aspect of this invention, a method for inserting a watermark 
in digital data is provided. The digital data into which the watermark is inserted 
If comprises a plurality of blocks of coded transform coefBdents, each block containing 
M one of a first type of coefficient, such as a DC coefficient, and a plm-ality of a second 
type of coefficient, such as AC coefficients. The method comprises the step of 
■ ~ appljdng a scrambling algorithm to each block in a predetermined pattern area of 
the digital data to insert the watermark in the digital data such that the resulting 
jc| watermarked digital data is format compliant. The scrambHng algorithm is applied 
iy by performing at least one of the following scrambling operations (i) for at least 
C some of the blocks in the pattern area, XORing magnitude bits of each of selected 
ones of the second type of coefficient with a pseudo random number, (ii) for a select 
number of pairs of blocks in the pattern area, swapping the plxirahty of second type 
20 coefficients from one block in each pair with the plurality of second type coefficients 
from the other block in the pair, (iii) for blocks in a first sequence of blocks in the 
pattern area, shuffling differential values of the first type of coefficient among 
blocks in the first sequence, or (iv) for blocks in a second sequence of blocks in the 
pattern area, shuffling select bits of the first tj^e of coefficient among blocks in the 
25 second sequence. 

The inserted watermsirk is preferably visible and obtrusive, and the 
scrambling algorithm is preferably reversible. 

Preferably, each of the second type of coefficient is coded by a code {R,,s(VJ) 
followed by s(F,)bits fi-om the least significant positions of the I's complement 
30 representation of V, , and if the scrambling algorithm is applied by performing at 
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least operation (i), then preferably the magnitude bits of each of selected ones of the 
second type of coefficient which are XORed Avith a pseudo random number 
comprises the s(y,) bits of that coefficient. 

If the scrambling algorithm is applied by performing at least operation (ii), 
5 then preferably the select nimiber of pairs of blocks to which operation (ii) is applied 
are constrained to lie within a predetermined number of rows of digital data. 

If the scrambling algorithm is applied by performing at least operation (iii), 
then preferably the differential values of the first type of coefficient are shuffled 
among blocks in the first sequence without changing the sum of the values of the 

10 first type of coefficient in the first sequence of blocks. 

If the scrambling algorithm is applied by performing at least operation (iv), 
then preferably the select bits of the first type of coefficient that are shuffled among 
J blocks in the second sequence are select left-most bits. 

Another aspect of the invention involves a method for inserting multiple 
£5 watermarks in an image which may include video fi:ame(s). The image comprises a 
plurality of blocks of coded transform coefficients, each block containing one of a 
first type of coefficient and a plurality of a second type of coefficient. The method 
comprises the steps of applying a first watermark to each block in a first 
predetermined pattern area of the image; and applying a second watermark to each 
20 block in a second predetermined pattern area of the image; wherein multiple 
watermarks are inserted such that the resulting watermarked image is format 
compliant, each watermark being applied by performing at least one of the 
scrambling operations (i), (ii), (iii) or (iv) described above. 

With this method, preferably the first predetermined pattern area comprises 
25 a central area of the image and the second predetermined pattern area comprises a 
peripheral ring surrounding the central area. 

In accordance with further aspects of the invention, any of the above- 
described methods or steps thereof may be embodied in a program of instructions 
(e.g., softwEU-e) which may be stored on, or conveyed to, a computer or other 
30 processor-controlled device for execution. Alternatively, any of the methods or steps 
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thereof may be implemented using functionally equivalent hardware (e.g., 
application specific integrated circuit (ASIC), digital signal processing circuitry, 
etc.) or a combination of software and hardware. 

In another aspect, the invention involves an apparatus for inserting a 
5 watermark in digital data that comprises a plurality of blocks of coded transform 
coefficients, each block containing one of a first type of coefficient and a plurality of 
a second type of coefficient. The apparatus comprises a processing device configured 
to apply a scrambling algorithm to each block in a predetermined pattern area of 
the digital data to insert the watermark in the digital data such that the resulting 
10 watermarked digital data is format compHant. The scrambling algorithm is applied 
a by performing at least one of the scrambling operations (i), (ii), (iii) or (iv) described 
: above. 

1==,, Preferably, the processing device is at least one of a microprocessor, an 

m application specific integrated circuit, or a digital signal processor. 
IM In accordance with another aspect of the invention, a method of selling 

sTii images (which may include video) over a network is provided. This method 
52 comprises the steps of maintedning a plurality of images on a network site 
|=* accessible by network browsing software; enabling a customer running the network 
browsing software to access the network site and to view a low quality version of 
20 each of the plurality of images; enabling the customer to select at least one image 
and to retrieve a watermarked version of each selected image; and transmitting a 
key to the customer in exchange for a specified payment, whereby the client can 
view or print a high-quality, watermark-free version of each selected image for 
which payment was received; wherein the watermark on each selected image was 
25 inserted in a predetermined pattern area of that image such that each resulting 
watermarked image is format compliant, the scrambling algorithm being applied by 
performing at least one of the scrambling operations (i), (ii), (iii) or (iv) described 
above. 

Preferably, the low quality version of each image available for viewing by the 
30 customer is a thumbnail version. 



patappAF 
Customer No. 20178 



4 



Express Mail Label No. EL700476615US 
Rev. 11/97 



AP119TP 



In addition, the key may include a descrambler and a printer driver, and may 
also be session-based. 

Another method of selling images (including video) over a network comprises 
the steps of maintaining a plurality of images on a network site accessible by 
5 network browsing software; enabling a customer running the network browsing 
software to access the network site and to view a low quahty version of each of the 
plurality of images; enabling the customer to select at least one image and to 
retrieve a version of each selected image having a jBlrst watermark in a first 
predetermined pattern area, such as a central area, and a second watermark in a 
|.Q second predetermined pattern area, such as a peripheral ring surrounding the 
i3 central area; and transmitting a first key to the customer in exchange for a first 
ii specified payment, with which the client can remove the first watermark of each 
2 selected image for which the first specified payment was received; transmitting a 
iSJ second key to the customer in exchange for a second specified payment, with which 
15 the cHent can remove the second watermark of each selected image for which the 
i*& second specified payment was received; the first and second watermarks having 
:i been inserted such that the each watermarked image is format compliant and by 
performing at least one of the scrambling operations (i), (ii), (iii) or (iv) described 
above. 

20 Other objects and attainments together with a fuller understanding of the 

invention will become apparent and appreciated by referring to the following 
description and claims taken in conjunction with the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1(a) shows an un-watermarked image. 

25 Fig. 1(b) shows the image of Fig. 1(a) with an obstructive visible watermark 

appHed in accordance with embodiments of the invention. 

Fig. 1(c) shows the image of Fig. 1(a) with a conventional "blended" visible 
watermark. 

Fig. 1(d) shows the image of Fig. 1(a) with corner logo visible watermark. 
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Fig. 2 is a schematic diagram of the block ordering used in embodiments of 
this invention, illustrated using a Y-Cb-Cr JPEG image with width 160 and 
chrominance sub-sampled by 2 horizontally and vertically. 

Fig. 3 is a schematic diagram showing the magnitude bits for a sequence of 
5 differential DC values. 

Fig. 4 is a flow chart illustrating the watermarking process in accordance 
with embodiments of the invention. 

Fig. 5 is a block diagram illustrating components in an exemplary image 
processing system that may be used to implement aspects of the invention, 
p Fig. 6 is a block diagram illustrating components of a cUent-server computer 

H system that may be used to implement aspects of the invention. 



m 



DESCRIPTION OF THE PREFERRED EMBODIMENTS 



A. Overview 

Mi 

Ml The present invention provides visible watermarking techniques which 

j£f prevent or discourage unauthorized use of the watermarked digital data and, in so 
p doing, solves various problems that arise in real and practical commercial imaging 

applications. An overview of the solutions provided by this invention are described 

below. 

Consider a still image, / , as shown in Fig. 1(a). This invention provides a 
20 keyed scrambling algorithm S such that r = S(I,K) is an image with the same 
dimensions as /, and is similar to / except in certain regions that form a 
distinctive pattern. In these regions comprising the visible watermark the 
scrambled image /' appears random or corrupted. The overall and intended effect 
is exemplified by the image in Fig. 1(b). The scrambling is secure in the sense that 
25 it is nearly impossible for an attacker, without knowing the key K , to recover the 
original image from Moreover, the scrambling is fully reversible, given the key. 
That is, a complementary descrambling algorithm £) exists such that 0(1', K) is 
exactly the same as / . The effects in the case of video are essentially the same — 
the scrambled video is viewable, but shows a distinctive pattern corrupting parts of 
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at least some of the video frames. For comparison, examples of two traditional 
visible watermarking schemes are also shown in Figs. 1(c) and 1(d). Fig. 1(c) shows 
a "blended" visible watermark created by simple alpha-blending, and Fig. 1(d) 
shows the simplest possible visible watermark — a logo in a comer of the image. 
5 For uncompressed raw images the desired effect shown in Fig. 1(b) can be 

achieved, for example, by XORing the pixel values within the pattern area with a 
keyed pseudo-random mmiber sequence. While this approach is acceptable for 
uncompressed images, it is not particularly useful for (lossy) compressed data. 
Compression introduces a complication: the approach of modifying raw pixels 
10 would require that the image first be decompressed, then the modifications applied, 
f! and then the image compressed again — a lossy process. In such a scheme it would 
{I be hard to ensiire that the original image can be recovered exactly upon subsequent 
descrambling. 

IS This invention provides a scrambler S and descrambler D for transform- 

15 coefficient-coded host signals (e.g., Huffman-coded JPEG images) to solve this 
problem. In accordance with the invention, a compressed image / is scrambled to 
produce I' = S{I,K) which is also a valid image in the same compressed format and 

|!r shows the distinctive pattern. (Note that since the description is directed to 
compressed images, the notation is not complicated with any additional qualifiers, 

20 such as c(I) to indicate compressed images.) A very advantageous feature of this 
format-compliant visible watermarking approach is that /' can be viewed by any 
image viewer capable of viewing / , and can also be passed through any 
compression-aware components of an imaging and/or network system. 

In some embodiments, the scrambling algorithm provides another desirable 

25 property as well: it introduces no coding overheads. That is, the size (in bytes) of Z' 
is exactly the same as that of / . In other embodiments, a variant scrambling 
algorithm is provided in which a moderate size increase is incurred to make the 
watermark appear more strongly. 
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B. JPEG Review 

The "Joint Photographic Experts Group" (JPEG) compression standard is a 
transform-based image compression/decompression architecture that is widely used 
for efficient storage and transmission of images. While the techniques of this 
5 invention may employed with other known transform-based 
compression/decompression formats, such techniques are specifically designed with 
the widely accepted JPEG standard in mind. Hence, some core concepts of JPEG 
are reviewed below, with emphasis on those pertinent to the visible watermarking 
techniques of this invention. 
10 A JPEG image consists of 8 x 8 blocks which have been transformed using 

the Discrete Cosine Transform (DCT), quantized using a table of 64 scalar 
quantizers, and then entropy-coded. The 8x8 blocks of the image are grouped into 
'? Minimum Coded Units (MCUs). Each MCU corresponds to an equal area of the 
f image, with the dimensions of this area being multiples of 8. The niimber of 8 x 8 
15 blocks from different color planes that comprise an MCU may be different, 
C depending upon sub-sampling, the details of which can be found in any of a number 
of JPEG references. For the piorpose of this invention, it is sufficient to summarize 
•« the structure of a JPEG image / consisting of N color planes as follows: 

1. / is a sequence of coded MCUs. There are h rows of w MCUs each. The 
20 MCUs are coded in raster order. 

2. Each MCU consists of m„ 8 x 8 blocks from color component number 
n(0<n<N). 

3. An MCU is coded by coding the blocks from color component number 0 
(in raster order within the MCU), then the blocks from component 1, and so on. 

25 It is useful to consider this JPEG image structure in a color component-wise 

manner. That is, look at the 8x8 blocks for each (potentially sub-sampled) color 
component separately, and in the relative order in which they would be found 
within the JPEG format. This simplifies the presentation as it is not necessary to 
refer to further details about MCUs. Let F„ ,^ denote the 8 x 8 block of quantized 

30 coefficients numbered A: from color component number n. Note that this notation 
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has absorbed the fact the k is not necessarily the raster index of the block, because 
of sub-sampling and MCU structure. Here, 0<«<A^and 0<k<whm^. This 
notation is illustrated in Fig. 2, which shows the three color planes of a typical 
JPEG image in Y, Cb, Cr format where the chrominance components (Cb and Cr) 
5 are sub-sampled by 2 both horizontally and vertically. The thicker lines separate 
MCUs while the thinner lines separate the 8x8 blocks within the Y-component of 
the MCU. Each Cb and Cr MCU consists of only one 8 x 8 block. The blocks lying 
on the watermark pattern are shown shaded. The image width is 160, and the 
MCUs are 16 x 16 areas of the image. 
10 Let F^i^iz) denote coefficient number z (0 < z < 63) in the zig-zag order used in 

J JPEG. The first coefficient, F„ ,^(0) , is the DC coefficient which is the scaled average 
of all the pixel values in the 8 x 8 block. The DC values within each color plane's 
:7 blocks are differentially coded in JPEG to exploit correlation across blocks. Thus, 
for > 0 , the value coded in the bitstream is (0) - (0) . Let £)„ ^ denote this 

H differential DC value, keeping in mind that £>„ „ = -^„,o(0) • 

U 

Q The JPEG standard allows either HufBnan coding or arithmetic coding to be 

O 

used for entropy coding. While the techniques of the present invention may be 
employed with any known entropy coding, the following discussion focuses on 
Huffman-coded images since Huffman coding is the chosen alternative most of the 

20 time. In a Huffman-coded JPEG image, differential DC values D„j^ are coded as 
follows. Given a value v to code, let s(v) denote the "magnitude category^ of v, 
which is the nimiber of bits in |v| up to the most significant 1. For example, 
s(0) = 0, s(l) = s(-l) = l,s(3) = s(2) = ^(-2) = ^(-3) = 2, and so on. The JPEG bitstream 
consists of a Huf&nan code for ^(£>„ J , followed by s(D„j^) least significant bits from 

25 the I's complement representation of D„j^ . The DC Huffman coding tables may be 
different for different color components, or some of them may be the same. 

The coding of AC coefficients (1 < z < 63) also loses the magnitude category 
function sQ) . Additionally, the frequently occurring situation of long runs of zeros 
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within AC coefficient sequences is also exploited in the JPEG format. For the 
sequence of AC coefficients in a block F^ j^ consider the representation: 

Here, Rq is the length of the nm of zeros leading to the first non-zero 
5 coefficient (which has the value V^). This is followed by a run of zeros and then 
another non-zero value (F;), and so on. The JPEG bitstream consists of Hufi&nan 
codes for pairs of (R„s(V,)). Each such code is followed by siV,) additional bits 
taken from the least significant positions in the I's complement representation of 
v.. The AC Huffman coding tables may also be different for different color 
10 components. 

C. The Scrambling Algorithm 

The goal of the scrambling process is to "corrupt" those 8x8 blocks which lie 
in the pattern area, while making sure that the decoded image will not be corrupted 
outside the pattern area and also to ensure that this corruption can be undone by 
the descrambler. 

f Let pin,k) be a binary function that is 1 if and only if (block number k 

from component number n ) lies in the pattern area. The function p (and hence the 
pattern) need not be secret, and therefore may be available to any attacker. In 
some embodiments, the scrambling algorithm efficiently produces a JPEG image 

20 r = S(I,K) such that I' and / are exactly the same for MCUs where p(i,j) is 0, 
and are distinctly different where pii,J) is 1. In other words; the blocks where p is 
1 are scrambled by S. The descrambler D reverses the scrambling completely, 
with the correct key. 

The scrambhng algorithm comprises a set of operations. In one embodiment, 

25 all of these operations are applied while in other embodiments a subset of these 
operations are applied. In a practical implementation, the selected operation(s) 
is/are applied in a single pass through the image. For clarity, each of these 
operations is separately described below. The scrambling and descrambling 
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algorithms preferably use a common pseudo random nimiber generator (PRNG) 
whose seed is the secret key. PRNG may also be used to generate masking values 
as well as random permutations and selections. To keep the notation simple, the 
subscript pj^^ is used to indicate that a value, permutation, or selection has been 
5 obtained by invoking the PRNG . 

C.l. AC Masks 

For each block for which p(n,A:) = l, the magnitude bits of the AC 
coefficients are masked by XORing with a value obtained from the PRNG . As 
discussed above, the AC coefficients are coded by grouping them into rim-value 

fl pairs, iR,V), and the coded bitstream consists of a Huf&nan code for iR„s(V,.y) 

O 

followed by 5(F) bits from the least significant positions of the I's complement 

O 

^- representation of . Referring to these s(V,) bits as B , the AC masking operation 
^ comprises replacing B with B®Xp^a, where Xpj^^ represents random values. 

This maintains format compliance and does not change the size of the coded image. 
M It will be apparent to one skilled in the art that the descrambler can reverse this 
; - operation by XORing again with Xpg^Q . 

C.2. AC Swaps 

Since the AC coefficients are coded independently from block to block, the 
entirety of AC coefficients can be swapped across any two blocks for which pis 1. 

20 That is, if p{n,k^) = p{n,k^) = \ , then F„ (1,...,63) can be exchanged with 
F„ (1, . . .,63) . This would not change the bit rate at all. In fact, if the AC Huffman 
tables for two color components are the same (as is usually the case for the 
chrominance components Cb and Cr in JPEG images), then the swaps can be done 
across these components too. 

25 In practice, it may be better to perform the swaps only within a small 

number of rows to minimize the buffering reqmrements of the scrambler and 
descrambler. Thus, in one embodiment the AC-swap operation comprises selecting 
(using the PRNG ) a number of pairs of blocks F„^ fc^ , F„^^^ constrained to lie within a 
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buffered amount of MCU rows and such that both the selected blocks lie in the 
pattern and use the same AC Huffman table. For each such pair, all the AC 
coefficients are exchanged. 

C.3. DC Shuffles 

5 The previous two operations modify the AC coefficients. In practice, however, 

modifications to DC coefficients has far more impact upon watermark visibility than 
modification of AC coefficients. Accordingly, this and the next sub-section describe 
operations that modify DC coefficients. 

Consider (within color component niimber n) a maximal consecutive 
Jl| sequence of / blocks 

a 

each lying on the pattern. This is maximal in the sense that 
p{n,k-\) = p{n,k + l) = 0 and p(n,i) = \ for k<i<k + l. The DC-shuffle operation 
i=* corrupts the DC value of each block in the sequence, ensuring that the DC value for 
t§ the first block after the sequence remains unchanged, and that the bits required 
3S remain the same, by shuffling the differential DC values. 

Recalling that the notation D^j^ is used to denote the differential DC value 
for F„^, the DC value for the block after the sequence is given by equation (1): 
*+/ 

i=k 

k+l 

20 Thus, as long as the sum ^D„ , does not change, the individual £>„ , values 

may be fi-eely changed without affecting the DC values of the first block after the 
sequence. Moreover, if the changes are made by simply permuting the Z)„^ , then 
the bit rate will also be exactly the same. 

In one embodiment, the DC-shuffle operation generates a random 
25 permutation cj^^^ and applies it to the sequence of D„ , values, k<i<k + l . Note 
that the differential DC value of the first block after the sequence also participates 

patappAF 

Customer No. 20178 -, 9 



AP119TP 



in this permutation, but the net result is that its DC value (actual, not differential) 

is not changed. 

3.4. DC Bit Shuffles 

Fig. 3 shows the sequence of DC values identified in the previous sub-section. 
5 Here, the first I blocks are the ones to be corrupted, while the block has 
p{n,k + l) = 0 and needs to be left unmodified upon decompression. The figure 
shows 1 and 0 bit values starting from the rightmost column (nimibered 0) up to the 
column number corresponding to the magnitude category of the differential DC 
value. Thus, in the row £>„,,all colimins numbered s{d„^ and greater are marked 
W by X to indicate that these bits are not present in the bitstream. 

Now, consider the bit-wise composition of the D values in this sequence of 
blocks, k<i<k + l . For simplicity the subscript n is dropped. Let 5,. denote s{d„^ ) 
and let {j\ (O < j <s^) denote the s. bits of the I's complement representation of 
D„.. Then, 

15 A . = [6, - 1) - 1] [2^' - 1] + X 0')2^ (2) 

Here, the first term just makes sure that if the left-most bit (i.e., bit number s.-\) 
is 0, then the appropriate negative number is obtained by the subtraction operation 
2*' - 1 . Equation (2) can be rewritten as: 

D„^=\- r- + 6, is, - \% X T'-' - 1]+ 2 b, . (3) 

v=o 

20 Observe that because of the I's complement representation, the left-most bit 
b^Si - 1) of each Z)„ , has a disproportionate weight. 

By substituting the value of £>„ , from equation (3) into equation (1) it can be 
seen that the DC value of the first block after the sequence, i.e., F„^^,(0), is a 
constant plus a weighted sum of all the Z>,(y) values, with the weight being 2^ for 
25 bits other than the left-most ones. The weight of each left-most bit is 3 x2'"^ - 1 . 
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Thus, bit-plane permutation can be applied to these bits, while keeping 
F„^^,(0) unchanged, as long as care is taken not to mix left-most bits with non-left- 
most bits. Let q denote the maximimi value of s,. , for ^ < / < ^ + / . Then, for each 
bit-plane 7, (0< j<q), apply two pseudo-random permutations within the bXj) 
5 values: C7j.^e(7)and ;r„(y). In the first permutation, <7/,^g (y) , only those bits 
from bit-plane J participate which are the left-most bits. That is, only the ^,0) for 
which 5^-1 = 7 participate in cr^^gO). The non-left-most bits participate in the 
second permutation. That is, only the bXj) for which s^-\> j participate in 
^prngU)- 

yO It can be seen that applying these Iq permutations (two disjoint 

% permutations ap^g{j)3ind Jipj^^ij) on each bit-plane) ensures that the DC values 
within the pattern area are corrupted securely, while the DC value of the first 
^ following block is untouched. Moreover, the coded size remains exactly the same as 
hk it was prior to this operation. 

!'f Four operations that are applied with the scrambling algorithm of the 

present invention have now been described. The descrambler is straight forward in 
that it merely applies the inverse operations. The scrambling operations have been 
designed such that they are reversible. The determinations made dxiring 

20 scrambling (such as the s{D„^) values) are not affected by the scrambling and hence 
can be repeated by the descrambler with identical results. 

C.5. A Variant with Modest Size Increase 

The previous four operations have been designed to ensure that the 
compressed size of the scrambled image 7' is exactly the same as that of the 
25 original image I . Sometimes this may not be a strict reqmrement, and it may be 
acceptable to incur a modest increase in size. In such situations, the visibility of the 
watermark can be increased considerably by one additional operation. Here, a 
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pseudo random number can be added to each DC value within the pattern area. 
Thus, if p(n,k) = 1 , then set 

where X^^g can be positive or negative. Note that this is a reversible operation, 
5 when care is taken to do the addition modulo the limit on the magnitude of DC 
differential values. Since each such actual DC value is being modified, the 
differential DC values will also change. Moreover, the changes are not guaranteed 
to maintain the compressed size. Indeed, since the "randomness" of the DC 
coefficient differentials is being increased, the compressed size will increase. 
However, by placing limits on the additive value of X^^^g ' tl^is size increase can be 
kept small. 

Q 

D. Flow Chart Summary of Watermarking Operations 

^ The flow chart of Fig. 4 summarizes and illustrates, by way of example, the 

M; watermarking process in accordance with embodiments of the invention. As 
illustrated in the figure, in step 41 blocks of decoded transform (e.g., DCT) 

jSf coefficients of the image/frame to be watermarked are obtained. Next, in step 42, 
those blocks in a predetermined pattern area of the image/firame are identified. In 
step 43, an AC masking operation is applied to the AC coefficients of blocks in the 
pattern area. A second watermarking operation is performed in step 44. There, a 

20 certain number of pairs of blocks in the pattern area are selected and the AC 
coefficients are swapped across the block pairs. In steps 45 and 46, watermarking 
operations are performed on the DC coefficients in a sequence of blocks in the 
pattern area, without effecting the DC coefficient in the first block after the 
sequence. In step, 45, the differential DC values among blocks in a sequence of 

25 blocks in the pattern area are shuffled, while in step 46 DC bits are shuffled among 
DC coefficients in a sequence of blocks in the pattern area. None of these four 
operations increase the bj^e size of the image. If this is a strict requirement, the 
process proceeds through the "no" terminal of decision block 47 to generate the 
watermarked image/firame. If, however, a modest size increase is acceptable, the 

30 process can proceed through the "yes" terminal of block 47 to step 48 where another 
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watermarking operation is applied. There, a pseudo random number is added to 
each DC coefficient in the pattern area, then the watermarked image/frame is 
produced. 

It will be noted that the process illustrated in Fig. 4 employs all of the size- 
preserving watermfirking operations. While this a preferred embodiment, it is not 
the only embodiment. In other embodiments, various subsets of these operations 
may be applied to achieve various degrees of effectiveness, bearing in mind that the 
more operations that are applied the more effective the resulting watermark. 

E. Video Data 

In the case of DCT-coded video data, such as in the various MPEG formats, 
motion prediction causes a problem. Applying the scrambling algorithm described 
above to MPEG video may result in corruption of blocks outside the pattern (for 
which p is 0) because their motion vectors may be predicted from areas of other 
frames which have been corrupted. Thus, the distinctive pattern would show in the 
video, but there would likely be additional corruption outside the pattern. This 
additional corruption may be small or extensive, depending on the video sequence 
and block-matching algorithms used during coding. 

One way to alleviate this problem is to only apply the scrambling to bi- 
directional coded frames (B-frames), since these frames are not used for predicting 
any blocks or other frames. Typically, B-frames comprise a significant fraction of 
all frames, and in such cases, the pattern shows up distinctively-albeit in a 
flashing manner. But sometimes coded video sequences do not contain any B- 
frames. In such cases, the only frames which can be properly corrupted (that is, 
without propagating the corruption to arbitrary areas on other frames) are those 
uni-directionally predicted frames (P-frames) which are immediately before a non- 
predicted frame (I-frame). 

The terms "image" and "images" as used in the claims in intended to include 
video, in addition to still image(s). 
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F. Implementations 

The techniques of the present invention may be conveniently implemented in 
software which may be run on an image processing system of the type illustrated in 
Fig. 5. As illustrated in Fig. 5, the system includes a central processing unit (CPU) 
5 51 that provides computing resources and controls the computer. CPU 51 may be 
implemented with a microprocessor or the like, and may also include a graphics 
processor and/or a floating point coprocessor for mathematical computations. 
System 50 fxirther includes system memory 52 which may be in the form of random- 
access memory (RAM) and read-only memory (ROM). 

10 A number of controllers and peripheral devices are also provided, as shown in 

: Fig. 5. Input controller 53 represents an interface to one or more input devices 54, 
such as a keyboard, mouse or stylus. There is also a controller 55 which 

^ communicates with a scanner 56 or equivalent device for digitizing docimients 
including images to be processed in accordance with the invention. A storage 

15 controller 57 interfaces with one or more storage devices 58 each of which includes a 

;T storage medium such as magnetic tape or disk, or an optical medium that may be 
used to record programs of instructions for operating systems, utilities and 

O applications which may include embodiments of programs that implement various 
aspects of the present invention. Storage device(s) 58 may also be used to store 

20 data to be processed in accordance with the invention. A display controller 59 
provides an interface to a display device 61 which may be a cathode ray tube (CRT), 
thin film transistor (TFT) display or video player. A printer controller 62 is also 
provided for communicating with a printer 63 for printing docxunents including 
images processed in accordance with the invention. A communications controller 64 

25 interfaces with a communication device 65 which enables system 50 to connect to 
remote devices through any of a variety of networks including the Internet, a local 
area network (LAN), a wide area network (WAN), or through any suitable 
electromagnetic carrier signals including infrared signals. 

In the illustrated system, all major system components connect to bus 66 which 

30 may represent more than one physical bus. 
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Depending on the particular application of the invention, various system 
components may or may not be in physical proximity to one another. For example, 
the input data (e.g., the input image to which S or D is to be applied) and/or the 
output data (e.g., the output image to which S or D has been applied) may be 
5 remotely transmitted from one physical location to another. Also, programs that 
implement various aspects of this invention may be accessed from a remote location 
(e.g., a server) over a network. Such data and/or programs may be conveyed 
through any of a variety of machine-readable medium including magnetic tape or 
disk or optical disc, network signals, or any other suitable electromagnetic carrier 
10 signals including infrared signals. 

While the present invention may be conveniently implemented with software, 
a hardware implementation or combined hardware/software implementation is also 
^ possible. A hardware implementation may be realized, for example, using ASIC(s), 
■0 digital signal processing circuitry, or the Uke. As such, the term device-readable 
15 medium farther includes hardware having a program of instructions hardwired 
C- thereon. With these implementation alternatives in mind, it is to be imderstood 
S that the figures and accompanying description provide the functional information 
^3 one skilled in the art would require to write program code (i.e., software) or to 
fabricate circuits (i.e., hardware) to perform the processing required. 

20 G. Applications 

The visible watermarking techniques of the present invention, which are 
designed to prevent or discoin-age unauthorized use of the digital data, may be 
employed in a variety of real and practical commercial imaging applications. One 
such application is an Internet-based "image shop" which sells high-quality 

25 professional art that customers may purchase remotely and print. Such an 
arrangement is schematically illustrated in Fig. 6. The images offered for sale are 
maintained as image files 71 on a server 72 that is accessible through a network as 
the Internet 73. The customer, using a client computer 74, runs Internet client 
software, such as a web browser 75, to access the web site of the "image shop," to 

30 browse thxmibnails of the artwork and to retrieve the ones in which (s)he is 
interested. When (s)he retrieves a high-quality image, the server 72 sends a 
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watermarked image which (s)he can view. The retrieved image, while viewable, is a 
much inferior version because of the obtrusive visible watermark. With the correct 
key, the customer may be provided with an integrated descrambler and printer 
driver, which will print the image free of the watermark. While key management 
5 issues will vary depending on the requirements of the overall system, the key in the 
above system, for example, can be session-based, to be retrieved securely by the 
client software upon the customer's payment. 

To facilitate the transactions, the server 72 contains appropriate watermark 
removing tools 76, in accordance with this invention, which may be in the form of 
10 software and may include keys, descramblers and printer drivers. The images may 
Q be watermarked either before or after they are uploaded to the server 72; if after, 
T the server 72 would also include watermarking tools in accordance with this 
r:; invention. The server 72 also contains appropriate software 77, which verifies 

'■0 payment and interacts with the watermark removing tools so that the appropriate 

ifi 

15 tools can be transmitted to the client 74 for removal of the watermark(s). Software 
i2 77 also preferably includes software for ensuring secure transmission of images, 
Ji;; payment and watermark removing tools. 

K The advantage of using the visible watermarking scheme of this invention in 

the above system, as opposed to simple encryption of the whole image, is that the 

20 client software can display the watermarked image. Moreover, standard 
technologies for retrieving content over the Internet can be used, with the 
knowledge that cached copies of the artwork (in the network as well as in the 
client's local disk cache) will only be of marginal value because of the obtrusive 
watermark. 

25 For video, the sample application could be delivered over satellite or cable 

networks. Again, the watermarked video would be viewable to give the flavor of its 
content without revealing all of it. Paying customers may retrieve the key(s) 
needed to descramble and view the un-watermarked video. 

Another appHcation involves watermarking the same image/video multiple 

30 times with different patterns and keys. For example, a first watermarked pattern 
area obscures a small central area. A second watermarked pattern area obscures a 
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larger peripheral ring. A user pays a certain amount of money to get the key to 
remove the watermark on the peripheral ring. The user then pays an additional 
amount of money to remove the watermark on the central area. 

H. Effects 

As the foregoing description demonstrates, the present invention provides 
visible watermarking techniques for digital images and video in transformed-based 
compressed formats. The insertion algorithm reversibly but securely scrambles 
parts of the digital data to obstrusively mark the image. Full access to the image is 
obtained by procuring the correct key to completely remove the watermark to 
recover an image exactly the same as the original. The watermarked data is 
completely format-compUant, and in most of the techniques, the same size as the 
original. 

The scrambling and descrambhng algorithms are very efficient and operate 
on transform (e.g., DCT) coefficients obtained by decoding the entropy (e.g., 
Huffinan) codes. Advantageously, the algorithms do not require that the 
coefficients be de-zigzagged nor dequantized. In fact, the algorithms only examine 
the compact sequence of the non-zero coefficients within each block. 

While the invention has been described in conjunction with several specific 
embodiments, many further alternatives, modifications, variations and appHcations 
will be apparent to those skilled in the art that in hght of the foregoing description. 
Thus, the invention described herein is intended to embrace all such alternatives, 
modifications, variations and applications as may fall within the spirit and scope of 
the appended claims. 
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